Skip to content

Fix OpenLineage macro _get_logical_date#51210

Merged
mobuchowski merged 3 commits intoapache:mainfrom
astro-anand:fix-openlineage-get-lineage-af3
May 30, 2025
Merged

Fix OpenLineage macro _get_logical_date#51210
mobuchowski merged 3 commits intoapache:mainfrom
astro-anand:fix-openlineage-get-lineage-af3

Conversation

@astro-anand
Copy link
Contributor

@astro-anand astro-anand commented May 29, 2025

With the current implementation I was getting the error:

[2025-05-29,` 10:55:56] ERROR - Task failed with exception: source="task"
UnboundLocalError: cannot access local variable 'dag_run' where it is not associated with a value
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 838 in run

File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1130 in _execute_task

File "/usr/local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 408 in wrapper

File "/usr/local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 251 in execute

File "/usr/local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 408 in wrapper

File "/usr/local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 212 in execute

File "/usr/local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 235 in execute_callable

File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/callback_runner.py", line 81 in run

File "/tmp/airflow/dag_bundles/astro/main/dags/2025-05-27T19:59:28.0625152Z/dags/ml_ops/model_train.py", line 320 in train_and_evaluate_models

File "/usr/local/lib/python3.12/site-packages/astro_observe_sdk/metrics.py", line 32 in log_metric

File "/usr/local/lib/python3.12/site-packages/astro_observe_sdk/utils.py", line 39 in get_lineage_run_id

File "/usr/local/lib/python3.12/site-packages/airflow/providers/openlineage/plugins/macros.py", line 68 in lineage_run_id

Updated the openlineage macro _get_logical_date to set the dag_run if it is retrieved from context instead of task_instance. context is a typeddict, so hasattr(context, 'dag_run') will always return None. The correct way of accessing the dag_run key of the context variable is with context.get('dag_run'). I updated the control flow to be if-else because the context variable should always have the dag_run key. Please correct me if this is an incorrect assessment.

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@astro-anand astro-anand requested a review from mobuchowski as a code owner May 29, 2025 15:59
@boring-cyborg
Copy link

boring-cyborg bot commented May 29, 2025

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@mobuchowski mobuchowski merged commit 6041b77 into apache:main May 30, 2025
70 checks passed
@boring-cyborg
Copy link

boring-cyborg bot commented May 30, 2025

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

sanederchik pushed a commit to sanederchik/airflow that referenced this pull request Jun 7, 2025
@astro-anand @sanederchik
Fix OpenLineage macro _get_logical_date (apache#51210)
c6ee7dc 
* change hasattr to get

* update to else
@eladkal eladkal mentioned this pull request Jun 15, 2025
Closed
jose-lehmkuhl pushed a commit to jose-lehmkuhl/airflow that referenced this pull request Jul 11, 2025
@astro-anand @jose-lehmkuhl
Fix OpenLineage macro _get_logical_date (apache#51210)
a6ea632 
* change hasattr to get

* update to else
@kacpermuda kacpermuda mentioned this pull request Dec 8, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mobuchowski mobuchowski mobuchowski approved these changes

Assignees

No one assigned

Labels

area:providers provider:openlineage AIP-53

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@astro-anand @mobuchowski